Skip to content

Enforce CPU+memory limits on Actor containers and stream systemInfo events to runs - #28

Merged
Pijukatel merged 15 commits into
v2-poc-requirementsfrom
claude/actor-runtime-resource-management-nmays9
Aug 26, 2026
Merged

Enforce CPU+memory limits on Actor containers and stream systemInfo events to runs#28
Pijukatel merged 15 commits into
v2-poc-requirementsfrom
claude/actor-runtime-resource-management-nmays9

Conversation

@Pijukatel

@Pijukatel Pijukatel commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

What

Actor containers now start under hard cgroup limits for both memory and CPU, derived from the run's memoryMbytes. The runtime measures each running container's real CPU/memory over the Docker API and pushes platform-shaped systemInfo events to the Actor over a websocket, and five new env vars tell the SDK where to dial and what its grant is. POST /actor-runs/:runId/abort gains an opt-in ?gracefully= that sends an aborting event before stopping the container. Two sample Actors, one per SDK, print the whole picture from inside a run.

Why

The runtime capped memory and nothing else, and set APIFY_IS_AT_HOME=1 while injecting neither an events-websocket URL nor a memory var. The SDK inside the container therefore picked its platform event manager, found no socket, logged one debug line and emitted nothing — so Crawlee's autoscaled pool ran blind: no CPU signal at all, and a memory ceiling guessed from the host's total RAM rather than the run's grant. The Actor scaled up until the kernel OOM-killed it, which is exactly the failure a developer comes here to avoid reproducing the slow way on the platform. Meanwhile CPU was unbounded, so one runaway Actor could starve the other four the documented scale budget allows.

What changed

  • CPU limit as a CFS quota. HostConfig.CpuPeriod: 100000 + CpuQuota: round(cores × 100000) at the platform's own memoryMbytes / 4096 cores ratio, floored at the daemon's 1000µs minimum. NanoCpus is deliberately never used: moby's verifyPlatformContainerResources rejects NanoCpus > NumCPU × 1e9 outright, which would turn "warn, never clamp" into "cannot run at all" for an over-capacity request.
  • Warn, never clamp. A request exceeding the host's reported capacity writes a warning naming both figures into the run's own log (where apify call shows it), then applies the requested limits verbatim. Capacity that can't be read is treated as unknown — never as zero, which would warn on every run.
  • Run telemetry. A per-run sampler polls one-shot Docker stats once a second and emits eight-field systemInfo frames: cpuCurrentUsage as percent of one core (the convention apify-sdk-python divides by APIFY_DEDICATED_CPUS), memMaxBytes as the configured limit rather than an observed peak, memory excluding page cache the way docker stats does, and isCpuOverloaded as a strict usedCores / grantedCores > 0.95. A stats body missing a field skips that tick whole rather than emitting a partial frame — a seven-field frame fails Python-side validation and a NaN would poison the running averages for the rest of the run.
  • Events channel. GET /actor-runtime/events/:runId upgrades on the existing API port. It carries no authentication (a local single-operator dev tool); isolation is structural instead — the fan-out module has no broadcast set, so a socket only ever receives frames for the run id in its own path. Unknown or already-terminal run ids complete the upgrade and then close 1008 rather than refusing the handshake, because apify-sdk-python treats a failed first connection attempt as fatal to Actor.init(). Normal run end closes 1000.
  • Env vars. ACTOR_EVENTS_WEBSOCKET_URL / APIFY_ACTOR_EVENTS_WS_URL and ACTOR_MEMORY_MBYTES / APIFY_MEMORY_MBYTES are each set byte-identically — the two SDKs resolve those aliases in opposite precedence order, so any divergence would size JS and Python differently from the same run. APIFY_DEDICATED_CPUS has no JS counterpart and exists so Python stops dividing by an assumed 1.
  • Graceful abort. ?gracefully=true publishes one {"name":"aborting","data":{}} frame, waits a fixed 30s window, then stops the container. Omitted or false is byte-for-byte today's behavior. A second graceful abort during an active window joins it rather than cutting it short; an explicit hard abort escalates. persistState is never server-sent — both SDKs generate it on their own timer.
  • Sample Actors. sample_actor_resources_ts and sample_actor_resources_py print the grant they were given and then one line per systemInfo event, making limits → sampler → websocket → env vars observable from a single apify call. They read the grant from different places, which is a property of the SDKs rather than a choice: the JS SDK re-emits each frame verbatim so a JS Actor reads memMaxBytes off the event, while the Python SDK maps the frame onto a usage-only structure with no total, so the Python sample reads Actor.configuration.memory_mbytes. Neither makes outbound requests, so unlike the crawling samples they run offline once their base image is pulled.
  • Docs. requirements/actor-driver.md, api.md and system.md updated in lockstep with the contracts above.

Proof it works

On the merged branch, pnpm run build, pnpm run lint, pnpm run format:check and pnpm test are green: 523 tests across 42 files. Of those, this branch adds 74 tests across 4 files (the base contributes the rest, 449 across 38 at cda7e9a); no existing test was weakened or removed on either side of the merge.

The regression tests worth knowing about, each written against a reproduced failure:

  • A malformed websocket frame from any container used to take down the whole runtime process (no 'error' listener on accepted sockets); graceful shutdown used to hang forever while any Actor was connected, because closeAllConnections() does not reach upgraded sockets.
  • sampler.stop()'s wait on an in-flight stats call is bounded, so an unresponsive daemon can't strand a run as RUNNING with its timeout timer already cleared; both grace-race timers are cleared once settled.
  • Fake-timer coverage for the 30s abort window, the 1s cadence, the > 0.95 threshold on both sides, memMaxBytes constancy, the skip-and-recover path for incomplete stats, and two-concurrent-run isolation in both directions.

Both sample Actors were verified against the real SDKs rather than by inspection: the TypeScript one type-checks and builds against apify@3.7.2, and the Python one imports cleanly against apify@4.0.1, where feeding this runtime's exact frame through the SDK's own to_crawlee_format yields cpu_info.used_ratio == 0.8 and 384.0 MB for a 1024 MB / 0.25-core run — the numbers the design predicts.

Real-daemon behavior (docker inspect showing the computed quota, emitted numbers tracking docker stats) is the one thing this suite can't assert — there's no Docker socket in CI.

Worth knowing before merge

Setting ACTOR_EVENTS_WEBSOCKET_URL makes this endpoint a hard dependency for Python Actors: a failed first connection raises out of Actor.init(). The 1008-after-upgrade policy keeps rejections survivable, and the existing Python sample-Actor e2e run becomes the canary — but the blast radius is real. JS Actors have no client-side reconnect, so the server must not drop a healthy run's socket for any reason short of run end or shutdown.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BeN7rnNX67yNdUWsRMA7Sp

claude added 15 commits August 25, 2026 13:54
…s to Actor runs

Actor containers now get a hard CFS CPU quota derived from the run's memory
grant at the platform ratio (memoryMbytes/4096 of a core, CpuPeriod/CpuQuota
encoding, 1000us protocol floor) alongside the existing memory cap. Requests
exceeding host capacity (per docker.info) warn in the run's own log and are
applied verbatim, never clamped.

The runtime now samples each running container's real CPU/memory once a second
(one-shot docker stats, stopped before container removal) and pushes
platform-shaped systemInfo frames over a new per-run websocket at
GET /actor-runtime/events/:runId on the existing API port (ws, noServer
upgrade). The endpoint is unauthenticated by design; isolation is structural -
each socket subscribes only to its own run's channel, unknown/terminal run ids
get a 1008 close after the upgrade, normal run end closes 1000.

Run containers receive five new env vars (ACTOR_EVENTS_WEBSOCKET_URL /
APIFY_ACTOR_EVENTS_WS_URL, ACTOR_MEMORY_MBYTES / APIFY_MEMORY_MBYTES,
APIFY_DEDICATED_CPUS) so Crawlee's autoscaled pool inside JS and Python Actors
receives platform resource estimation instead of running blind.

POST /actor-runs/:runId/abort gains ?gracefully= mirroring the platform: it
emits one {"name":"aborting","data":{}} frame, waits 30s, then stops the
container; the default abort path is unchanged. persistState is never
server-sent.

Requirements docs (actor-driver.md, api.md, system.md) updated in lockstep;
unit+integration coverage for the CFS math, warn-not-clamp, sampler lifecycle,
envelope shaping, env contract, isolation, close codes, and graceful abort.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeN7rnNX67yNdUWsRMA7Sp
Review fixes on the resource-limits/events-channel work:

- Every accepted events websocket now has an error listener, and the upgrade
  handler contains connection-setup failures to that one socket - previously a
  single malformed frame from any container crashed the whole runtime process.
- Graceful shutdown terminates the events server's own clients before closing
  the HTTP server: closeAllConnections() never destroys upgraded sockets, so
  shutdown hung whenever an Actor was connected. The comments claiming
  otherwise are corrected and shutdown.test.ts now guards this for the events
  socket like it already did for log streams.
- docker-driver's startRun try/finally now also covers container.start(), the
  sampler creation, and container.logs(), so a throw in that window still
  stops the sampler and removes the container.
- abortRun derives its was-running decision inside the status transition's
  per-id mutex (new onBeforeTransition hook on transitionJobStatus) instead of
  a separate unguarded read that raced READY->RUNNING; the redundant registry
  read on the default abort path is gone.
- gracefully=true is now exercised over a real HTTP round trip in the
  integration suite.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeN7rnNX67yNdUWsRMA7Sp
…s.ts

- Comments that cited ephemeral design/review documents now cite the durable
  requirements docs (requirements/actor-driver.md, requirements/api.md) or
  carry the rationale inline - most importantly the events websocket's
  no-authentication note, which now points at requirements/api.md instead of
  an artifact that never ships with the repo.
- The shutdown-ordering rationale (why the events server must terminate its
  own clients before closeServer) now lives only in
  EventsWebSocketServer.close()'s doc comment; the other three sites defer to
  it in one line.
- events-channel exports follow logs.ts's domain-prefixed convention:
  subscribeEvents / markEventsTerminal; the import alias in runs.ts is gone.

No behavior change: comments, import lines, test titles, and renamed
identifiers only.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeN7rnNX67yNdUWsRMA7Sp
…ately

A repeat abort call while a gracefully-aborted run is still inside its
30-second window used to stop the container immediately, silently defeating
the window: the ABORTING status alone cannot distinguish "my transition just
landed" from "another caller's window is active". abortRun now captures that
distinction atomically (the pre-transition status reported inside the
status-transition mutex): a second gracefully=true call joins the active
window and returns the current record without re-triggering anything, while a
plain abort deliberately escalates to an immediate stop - the same behavior a
plain double-abort always had. The first caller's eventual finalization stays
a safe no-op under the existing terminal-status guard.

Also covers the over-capacity warning's memory-only and CPU-only arms with
dedicated unit tests and fixes a stale subscribe() comment reference.
requirements/api.md documents the join/escalate contract.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeN7rnNX67yNdUWsRMA7Sp
Comment-only cleanup: constraints formerly cited as bare criterion or
iteration numbers now state the guarantee inline or cite the matching
requirements/*.md section (graceful-abort join/escalate contract, the
501-vs-404 spec-table rule, the APIFY_PROXY_PASSWORD contract, the
flusher-gated log-stream race). No code, identifiers, assertions, or
behavior changed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeN7rnNX67yNdUWsRMA7Sp
…omments

Comment-only: the two test comments that still cited a bare criterion number
now cite requirements/api.md's Graceful abort section.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeN7rnNX67yNdUWsRMA7Sp
…dedupe

Final-review fixes on the resource-limits/events work:

- sampler.stop() bounds its wait for an in-flight stats() call with
  SAMPLER_STOP_GRACE_MS (mirroring LOG_DRAIN_GRACE_MS) - an unresponsive
  daemon could previously stall run finalization forever, leaving the record
  RUNNING with its timeout timer already cleared.
- memCurrentBytes now excludes the page cache the way docker stats does
  (usage minus inactive file bytes, cgroup v1/v2 aware), so autoscaling sees
  reclaimable memory as free instead of steadily inflating usage.
- The events websocket path regex rejects a trailing slash, matching the
  documented route exactly.
- The memory/CPU ratio helpers move from services/ to src/resources.ts so the
  driver no longer imports the services layer.
- Shared pollUntilTerminal helper replaces the near-duplicate poll loops in
  the log and events endpoints; the Docker stats test stub moves to
  test/unit/helpers/docker-stubs.ts.
- requirements/api.md's abort entry names the actually-mounted route; test
  titles describe behavior; previously untested branches (unmatched upgrade
  path, warning arms, poll edges) gain coverage.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeN7rnNX67yNdUWsRMA7Sp
…frames

A Docker stats body missing memory_stats.usage produced a seven-field
systemInfo frame (dropped by the Python SDK's validation) and then poisoned
the memAvgBytes running average with NaN for the rest of the run; a body
missing cpu_stats threw synchronously into an unhandled rejection. The
sampler now checks both readings for presence and finiteness and skips the
whole tick when either is unusable - no frame, accumulators and the CPU delta
baseline untouched - so one bad sample costs one tick, never the run's
telemetry.

Also: the graceful-abort integration suites move byte-identical from
job-lifecycle.test.ts (1168 -> 802 lines) into graceful-abort.test.ts, the
sampler-grace paragraph in requirements/actor-driver.md now states the
bounded-wait semantics the code actually implements, and a test comment
states its constraint directly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeN7rnNX67yNdUWsRMA7Sp
Two unit tests now reach the CPU snapshot guard's second clause (valid
total_usage, missing or non-finite system_cpu_usage - the cgroup v2 rootless
shape), asserting the tick is skipped and the next good sample recovers.
Three comments corrected: two now state their constraint directly, one
cross-file reference follows the graceful-abort suite to its new home, and
the split rationale no longer cites a policy the repo does not have.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeN7rnNX67yNdUWsRMA7Sp
… test comments

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeN7rnNX67yNdUWsRMA7Sp
Resolves four conflicts from the pnpm migration (#26) and the
Dockerfile-location work (#27):

- package-lock.json: accepted its deletion; ws and @types/ws are now
  carried by pnpm-lock.yaml, regenerated with pnpm install.
- package.json: kept both sides' dependency additions (json5, ws).
- docker-driver.ts and test-server.ts: kept both sides' imports; the
  driver-stub type import now names BuildContext and RunResourceSample.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeN7rnNX67yNdUWsRMA7Sp
`sample_actor_resources_ts` and `sample_actor_resources_py` print the
memory and CPU the runtime granted the run, then one line per systemInfo
event, so the whole path this branch adds - cgroup limits, the sampler,
the events websocket and the injected env vars - is observable from a
single `apify call`.

The two differ in where the grant comes from, which is a property of the
SDKs rather than a choice: the JS SDK re-emits each frame verbatim, so a
JS Actor can read `memMaxBytes` off the event, while the Python SDK maps
the frame onto a usage-only structure with no total, so the Python sample
reads `Actor.configuration.memory_mbytes` instead.

Neither sample makes outbound requests, so unlike the crawling samples
they run offline once their base image is pulled.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeN7rnNX67yNdUWsRMA7Sp
The comments this branch added were roughly 40% of its own new lines,
against 24-33% in the same files at base; they now keep the constraints
that would otherwise be easy to reintroduce (the ws error listener, the
close-before-shutdown ordering, the sampler's bounded stop) and drop the
retelling around them.

The requirements grew by 187 lines describing how and why - moby's
validation rules, docker-modem's timeout handling, pydantic's field
requirements. They describe what the runtime does instead, in 87.

The standalone resource-reporting sample Actors are gone; the existing
samples log their granted resources and each systemInfo event, which is
the same demonstration without a second pair of Actors to maintain.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeN7rnNX67yNdUWsRMA7Sp
@Pijukatel
Pijukatel merged commit 86988d3 into v2-poc-requirements Aug 26, 2026
2 checks passed
@Pijukatel
Pijukatel deleted the claude/actor-runtime-resource-management-nmays9 branch August 26, 2026 12:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants